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(54) Distributed client-based data caching system and method 

(57) A system and method for enabling data pack- 
age distribution to be performed by a plurality of peer cli- 
ents connected to each other through a network, such 
as a LAN (local area network). Each peer client can 
obtain data packages from each other or from an exter- 
nal server. However, each peer client preferably obtains 
data packages from other peer clients, rather than 
obtaining data packages from the external server 
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Description • * * - 1 . ■ \ . , x . . ........ ... -.^"rlv- * 

: .. . *■ ;. t - •. t ■„-. • ■ ; &t3*Z 2: w'u;. * ■ .■ •.. » •• v - ■ 

[OOOil : •••'■me present Mention relates to a di^itJOt^d ilierttiased data caching f^f^^-^S^f* 
system and methodofthe present invention e'nablS dala 'packag.es.to be served to a client throughjMle^le. "Offdett- 
SSLd system of peer clients which c^che the data packages, in orderto rnax.rn.ze eff-cency and speed. 

55^ S^!^^S« «Sr rhore computers; such as the Internet or ^^^^S " 
obtain data packages; such as documents: images, messages, data packages or other types of data, from remote 
stora nS Sen « not installed on the client computer itself, Instead, these * m .r *° ra ^ ', 
and operated through a remote-computer, known as a server computer or simply as a "server {m fee same vein, the 
£££!£mi L often termed only a "client"). The advantage of such a system is JM ^ *£«^J» 
p^W obtain data from any server on.the network. The disadvantage o the system •^^ w ^J^^ t 
SSSm on the hetwork td enable data tb be transmitted from the server to the ^J"*^***?^ 
evenly attributed between servers on the network, one server may become overwhelmed *"J* 
decreasing the speed and efficiency of retrieval. Thus, currently many networks cannot prov.de rapid and efhcent data 

retrieval due to the heavy demands placed upon the available bandw.dth. 

0003] Proxy servers are often installed to. conserve bandwidth on an Internet connector, or on connections to 
rther LANs (local area netwbfks). These pVoxy servers cache frequently accessed data, thereby reducng the load on 
r^serler and distributing demand for bandwidth more evenly across the network. Unfortunately, such proxy 
servrs are typically expensive to maintain. Furthermore, proxy aerv^ 

on the LAN has to be'separately configured in order to commun.cate wrth the proxy server 
tSSiM iJdeterministic; such that each client must be configured to commumcate wrth each proxy server - 

eeoaratelv- Thus; proxy servers have many drawbacks. ; . , = 

S a SIi solution would enable Int^ane^s to reap the benef rtsd the proxy server, wrtho^ 

• Smach^WaSwrthout fequiring any special installation or configuration. Furthermore, such a s^ution. would not 
S oSnSfc SSat each client could communicate with more than one server according to the loadon each 
e^v^nZhh^^ to the configuration of the client itself. Unfortunately., such a solution .. not currently avail- . 

jSos] Therefore, there is an unmet need for. and it would be highly useful to have, a debuted clients** hdata 
Sg system and method which stored and retrieved from a pluralrty of peer Cents, or caching. 

entities" yet which does not require any special cortfig'uration or installation of separate servers^ _ _ 
foS ^^e present invention is of a distributed dieht-based datacachihg system and method, which enable data 
,Te served a c ^through a flexible. non<feterministic disputed system of caching entities, m 
SfSSSd speed forservlngtt.edocumehftoWclient } 
to eaSeMhereby reducing ^ - " 

u60# SStoihe present invention: Were is provided a method for distributing data packages across ane - • 
wo* the neSteaSring an external servertor serving at least one 6M^^m;^r^.^^^ 
2S5 steps^ot the rhethcd beingw termed by a data processor, the^^ BjrJJP ^ l; 

SvldS a plurality rfpeer clients attacheb fo Network and a list of data packages be,ng stored by each of the ^ 

Sr^edarlcSge^ 
Tple^ 

the plurality of peer clients according to the entry for the data package. • 

• rnnrtni > Aiterhativelv the : list ^ of data packages' is stored on the external server. 

SL P .eS^ S2ably.=H alternatively the entry ^ the data package is absent from trfe list of data packages of 
2 S Je I c en the methi further comprises the steps of: <d sending a request 

he first peer client to at least one other peer client and (e) if a response message. ,s «^^^ f £j£ 
, rom the at leasl one other pe^ 


[0010] Preferably, the request message andthe response message are.transm.tted to the p urahty of peer cl.ente 
by broadcasting. Alternatively, the request message and tHe response message are transmrtted to the P^* * P?" 
clients by multicasting. Also alternatively, the request message and the response message are transm.tted to the plu- 
s ralitv of beer clients by polling each peer client individually. 

[001 11 Also alternatively and preferably, if the response message is not received from the at least one other peer 
client by the first peer client the method further cbmprises the step ofi (f) obtaining the data package by the first peer 
client from the external server. Preferably, the method further comprises the step of sending a response message by 
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the first peer client to the at least one other peer client substantially before the first peer client obtains the data package, 
from the external server. More preferably, the list of data packages is stored on each of the plurality of peer clients, and 
the method further comprises Jhe steps of: (g) receiving the response.qjessage from the first peer client by the at least 
one other peer client; and (h), altering the list of data packages being.stor.ed, by the. at least one other peer client for indi- 

5 eating the lccati.on'!bf v the data package accdrding to the response message. 

[0012] Alternatively, the list of data packages is stored on each of the plurality of peer clients, and the method fur- 
ther comprises the steps of: (g) receiving the response message from the first peer client by the at least one other peer 
client; and (h) altering the. list of data packages being stored by the.at least one other peer, client for indicating the loca- ; 
tion of the data package, according to a probabilistic-function. . 7 - 1 . . . 

io [0013]-. Preferably, the probabilistic function is performed according to, a setoff ■ 4 • 

Old location , v ' !r Pp(x) - l/(generation+l) 
New location ^yj'j^(xy^ l-l/(generation+l). 

wherein Pn(x) is a probability that the new location is substituted for the o.ld.location, Ro(x) is a probability that .the old. 
location is retained, and "generation" indicates how many times the location had been previously changed. ; . , ~ t , t . 
[0014] Also preferably, an upper limit is predetermined for a number of the plurality of peer clients. served substan-. 
tially.simultaneously by the at least one other peer client, such that if a number of the plurality -of. peer clients served, 

25 substantially simultaneously by the at least one other peer client is greater than the upper limit, the method further. com- . 
prises the step of: (d) sending a busy message from the at least one other peer client to the first peer client. > w 
[001 5] , Preferably, the external server is a Web server, and the plurality qfpeer clients is a plurality of Web browsers.. 
[0016] Also preferably, the external sewer is a BackWeb™ server, and the plurality of peer clients is a plurality of 
BackWeb 1 " clients. . ; •*.-< ■■ - : . • : 

so [0017]. preferably, the unique identifier for the data package is an MDSdigest of the data package, v v , .... -. :: ' r , *- ti 
[0018] According to still other preferred embodiments of the present i.nyentipn, the step of retrieving the data pack-, 
age is performed according to a protocol based on T f CP/IP. Preferably. protocol is HTTP. Alternatively and preferably^ 
the protocols FTP.- < ,• , „. . y . ...v ; , . .. • :* . :i •■; ■; i* --v:./ 

[001 9] Hereinafter, the term "protocol based on TCP/IP" includes^ny. such protocol, indudingbut npt f limited to.the 

35 HTTP (hypertext transfer protocol) and Fin? (file transfer protocol) protocols. . , . ; . . A , ' r , : ;! 

[0020] . : . Hereinafter, the term^data package" refers to t any discr^e, r identifjable unit of data,, including but not limited; 
to documents, im^ » . ... ... 

[0021] . Hereinafter, the term ."computing platform" refers to a particular, computer hardware systemor to a particular 
software operating system. Examples- of such hardware systems jnclude tC but are not : limited to, personal computers, 

40 (PC), Apple Macintosh ™ computers, mainframes, minicomputers and workstations, which are also nonHimiting exam: 
pies of data processors : for 'operating a software application under . an ^operating system. Examples of such software 
operating systems ; include, but are not limited to, UNIX, VMS. Unua,'MacQ.S T " > DOS, one of the Windows™ operating 
systems by Microsoft Corp. (USA), including Windows NT™, Windows .3.x™ (in yyhich "x" is a version number,. such as 
"Windows 3.1™"j, WrxJows95™ 'and Windows98™ ! , .. ' i .."> v-'i-j ■'*•• . 

45 [0022] For the present invention, a software application could be .written in a substantially suitable programming, 
language, which could easily be selected by one of ordinary skill in the art" The programming language chosen should 
be compatible with the operating. system according to which the software application is executed.. Examples of suitable 
programmi ng languages include, but are not limited to, C. C++ and Java. ... . v . : 

[0023]. . Hereinafter, the term "broadcast' may also, include M multicast";as well. . ; 

so [0024] . The invention is herein described, by way of example only, with reference to the. accompanying drawings, 
wherein: 

FIGS. .1 A and 1 B are schematic block diagrams of an exemplary basic system and method according to the present 
invention; .... • • 

55 FIGS. 2A-2E are schematic block diagrams of an exemplary request/response protocol and method according to 
the present invention;. .... ..... ^. ... 

FIG. 3 is a schematic block diagram of an exemplary preferred data-flow diagram according to the present inven- 
tion; , ] ; > - .. ' r - 
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.FiG: 4, is a flowchart of a method jor .operating,the,syst.em of the presenWnvention with Web-browsers:- and -. 
FIGS 5A and Ware'exemplary request anctaespfclsemessages according to-the-present-hwenltons^ tfiwnoc- , 

[0025] "me present invention is of. a distributed'dient-based data caching system. and^eihod?which. ehable data 
to be sewed to a client through a flexible. non-determiriistic'distributed systenf of caching : entities. ? in.ordar tomax.mize< 
efficiency and speed for servingthe datato thedient. The caching entities we peer clients which-servethe data to.each ' • 
other thereby reducing. the amount of bandwidth required to obtain data from an external server/ '■■ ■■■■ w ....... 

[00261 The system and method of the present invention enable clients to share data packages among themselves > 
across their local network neighborhood,. for,example within a LAN. thereby eliminating the.need.for.a.specahzed proxy.; , 
server Furthermore.. the network. traffic is.not significantly affected, since modem network architectures are welrsurted •, 
for peer-to-peer communications. .Most.curreotly.pRerating.networks have a star topology, using swrtch.ng.hubs. in 
which communication.betweer.;two.peers.does..not .affect simultaneous communication among other nodes on the net- 
work. Thus, the system of the present invention, overcomes ^drawbacks of a proxy .server, yet does not add sign* .cant , 
loads to the traffic on the network itself. " • • 

[00271 . For currently available client-server software applications knpwn.in the art. whenever a cl.ent requires a data . 
package the following algorithm is performed. First, the software application attempts to locate the data package locally, 
on the memory or on the disk or.disks of the client.,Then. if the data package.is.nof.fpund locally. the software.appl.ca.-, 
tion retrieves the data package from the appropriateiserver. • ■ . : .- • ■ ' '' ; 

[00281 • By contrast the operation of the,system pf.the present-inverrtion adds an intermediate step.. For the present 
invention the data package is not found locally, an attempt is made to retrieve the data package from a peer.cl.ent on ■ 
the local network "neighborhood" before resorting to.retrieving the data package from the server. ;; . 
[00291 - Thus for the system of the present invention, every client actually functions asacaching proxy Oncea d.ent,: 
eouires a data package. it queries all the hosts.which are actually peer clients, on the local network for that data pack, , 
Lge f^o neighboring peeient has the.data package, the client retrieves the data package from the f * 
2 usual. However. H a neighboring client already has the required data package, the request^ chent w.ll download ? 
this data package.from.the peer client tatlmthai) lr.om.the • • ' '-- 4 ^±1^21 ' 

[00301 The principles and operation of the distributed client-based data cachmg system according to the. present 
invention may be better.understood with referenced the drawings : and the accompanying description.., %■ , • 
OwT TRgure 1 A is.a schematic bloc*diagram:Pf;-an exemplary . system according to:the present '"vention. wb-le 
Fioure 1B is I flowchart of the operation of the system of Figure 1A. Figure 1 A shows a system 10 wh.c^ jndudes^a ; 
SLofpe^^ 

23X^*»Ur clients nib****.®***** r.20.and "peer cli ent 2" 22 are ^^e;u = 
of |JU and without intending to-be Wting.in any way. Each peercl.ent 12,s also connected to an externa, 
server 16 of some type by an external connecrton.lBf Although only one external server 16 is shown; a.pluralrtyof exter- ;; 
nauervers coTd laiso be implemented. External, served 6 js a dedicated server, in thesense that this server.has a pn- 
mary or aTS a substantially significantrolaas^server for data,packages; ; As shown for the purposes oM lustration;,; 
external connection 18 only .connects to local : network,1.4 at one point,: although multiple such ^M*™^^ 
couldialso be implemented (not shown), Inaddition-external connection 18 could also optionally connect. each-peer cfc ; 
ent 12 directly to server 16 (not shown). . ^ ■:. .< ■. .-^ ■ :j- 1 ' j 

[00321 - Tne operation of system 10 according to.the.present invention is illustrated, with reference . also to P'9ure.1B,. , 
in stej 1 peer client 12. such as peer.cfient 12 looks for;a.data package in the local memory or d.sk cache ..of; that par, 
ticular peeTdient 12. rt the desired.data package.is.not found on.the local disk cache, then in step:^peer Ghent 12 que- 
ries any other peer client(s) 12 on.local network l4;to,determine whether another peer d.ent^ has^partocular date 
package., For example, peer client 20 could query.peer client 22..to determine whether, peer d.ent 22 has the desired. . 
data package. In step 3a. if peer client 22 has the desired data package, .then peer cl.ent 20 obta.ns the data package 
from peer client 22. Alternatively., as shown in step 3b„i* peer .client 22 does not have the desired data pactegethen 
peer client 20 obtains the data package from server 16 through external connection 18. Thus,.every peer, cl.ent 1 2 s 
also potentially, a seryer which is internal to-local network ^, and hence could be described as an .nternal.server to. 
distinguish peerclient : 12 fromexternal 66rver 16.... • o.--.- • ;-' : • 

so [00331 Each peer.client W.could.also.bedescribed as a "caching entity" and the.data stored breach chent for serv- 
ing to other peer dients 12 . as "cached date^or "cached data packages". • • • - • • • 

[0034] ; A number of different possible embodiments of the system of the present invention can be implemented, of - 
which two illustrative embodiments are shown with reference toithe Figures below. Briefly., Figures 2A^2D illustrate an 
exemplary mbodiment of the system of the present invention for.implemehtatjon with the software application of Back: 
Wetf" (BackWeb Technologies Ltd.. Ramat Gan, Israel) on a local area network (LAN). Figures 4 and 5A-5B illustrate 
an exemplary embodiment of the system of the present invention for implementation with a Web browser software appli- 
cation on the Internet. . ■ ' •' ' ' ' ' • ' 
[0035] Figure 2A shows an exemplary: local network 24 which features a plurality of peer .dients 12 of which three 
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are showri.for the purposes of '"discussion only and without intending r t6ibe limiting in anyway. For the purposes- of dis- 
cussion only, suppose-a peer.client 26. labeled -A?, wishes to obtainNfeur, data packages "W", "X' ( "Y7 and "Z". None of 
these data packages are local to peer dient 26, which must therefore obtain these data packages from either another 
peer client 1 2 t as.an internal. server, or from an external server (not;shown).' Local area network 24 features two other .-- 
5 peer clients 12: peer.cnent28.:labeled "B", and peer client 30, labeled "C. Peer.client'26 must therefore f irst.communi- ■* 
cate a request to peer client 28 and peer client 30 to : see if the desired data packages are available at either location, 
and then peer client 26 must obtain these data packages from peer client 28 or peer client 30 if the desired data-pack-, ■ 
ages are available.. ■* . ■- . * ;-. * . *= .** ■.- . 

[0036] - Preferably, two protocols are used for communication be^eenjjeer clierrts on a'lccal area network (LAN), a 
jo data package-exchange' protocol and- a control protocol. Specifically, the data package exchange protocol is used to ■ 
transfer data packages between peer clients.'once the desired data package-has been located, and is described in 
greater detail with respect to Figure 2B below. The control protocol enables each peer client to efficiently build and •* ' 
maintain tables which describe the location of available data packages- across the local area network by exchanging 
messages. 

is [0037] Each peer client maintains two hash-tables which contain information about data package location: a local-' ^ 
data packages table and a network-data packages table. The local-data packages table is 'a hash-table of data pack- : - 
ages which reside on the storage medium or media of the' peer client riser). -The network-data packages table is a hash- •* 
table of data packages which reside on the storage medium or mfedia of other clients on the local, network. This table 
contains the local area network address of the peer client on which each data package is being stored. The size of this- : 1 

so hash -table is preferably limited in order to reduce memory consumption. More preferably; each entry in the table has a • 
time-stamp, such that older entries are purged when the size of the table exceeds the upper permissible limit ' - " ' 
[0038] In order to effectively identify the desired data package, preferably each data package has unique identifier > 
or "fingerprint" associated with it. More preferably, this unique identifier i's'an MD5 digest of the content of the data pack-'! • 
age (for a description of the MD5 specification, which is an industry standard and would therefore be obvious to one ! of- 

25 ordinary-skill in the art, see "RFC 1321" at hnp://dsj'ntemic.net/rfc/rfc^321!txt). ^ '' 

[0039] Once any peer client knows both the unique identifier and thelocatioh of the data package bh the local net- 
work, that client can then proceed to download the data package. -However; the peer client may not know the location '* ■ 
of the desired data package, in which case the client must follow £ control protocol according to the present invention - 
in order to determine, the* location of the desired data package and-to enable the client to build these hash tables with- ... 

30 respect to future attempts to locate a data package. • ■• : . . :r 

[0040] The control' protocol is used to provide each client wittv knowledge about" the' locations of data packages- 
across the local network. In the preferred implementation illustratecfcwith respect to Figures*2A-2D, control messages • 
are preferably sent and received as broadcast or multicast packetsTLbcal area networks such as Ethernet networks t- 
support broadcast or multicast packets such that air peer clients: on i a "iocal area network receive the broadcast or mul-' * 

35 ticast packets. Effectively, a single packet can be sent to all peer clients by using broadcast or multicast, thereby reduc- 
ing the amount of traffic on the network required as a result of transmitting the request message (see for example - 
Chapter 42,: "Broadcasting 1 and -Multicasting",- of TCP/IP ' Illustrateti'Vo'ltime^by W. Richard 'Stevens, "Addison-Wesley* ■ 
1994).' However,) optionally the system ' of the present invention could poll each peer client individually with a control' 
message for that peer client, although this is not preferred since such individually addressed messages would consume -: :■ 

<o excessive amounts of available bandwidth.- In such a situation, preferably polling would be restricted to a certain group** 
of peer clients as internal servers,- in order to reduce the amount of traffic on the local area network. ~ : . ■' •> * i = : 
[0041] :For the preferred implementation in which broadcast or multicast is used, more preferably, the decision to- 
select either IP multicast or broadcast is made according to the configuration set by the network administrator for- the 
local area network. IP multicast -is preferable in terms of load on the clients of the local network, but may not be supv ,* 

<5 ported on all platforms (operating systems). More preferably, the TTL or Time to Live maiy be configured. The TTL spec-- - 
if ies the number of routers a packet can cross before being dropped. Configuring the TTL enables data package sharing ' 
to be expanded across subnet boundaries. - ' . : • . " 

[0042] As shown with respect to Figure' 2B, the control protocol of thepresent invention preferably operates as fol- • 
lows. In step 1, peer client" A" from Figure 2A looks for a data package on the local storage medium or. media. In step 

so 2, since the data package was not found locally on : the medium ormedia of peer-client "A", peer client "A" must down : -* 
load the data package and therefore preferably murticasts (or alternatively broadcasts) a reqiyes^message:- A request " 
message preferablyxontains a protocol 1 identifying version number (PVN) for the control protocol of the present inven- 
tion and a list of MD5 digests of the needed data packages, as shown in Figure 2C .• 
[0043] Optionally and preferably, rf more than one data package is desired, a list of requested data packages is 

55 included.in the request message rather than a single MD5 digest. -in order to reduce the total number of request mes- * 
sages on the network. 

[0044] In step 3, the neighboring clients, shown as peer clients "B* and n C in Figure 2A, receive this request mes- 
sage and search tor the requested data package in their local-data packages hashrtable. A peer client which does not- 
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find the data package locally does not reply, as sfattrttftstep 4a, Otherwise, in step 4b thepeer-cliefrt^erids a^espxj/ise : 
message preferably, after waiting-. a short randor.4i.me>interval to determine whether. ano^ 
first More preferably.- the peer client does .not distributatheresponse message if another.cKentoesponded previously/in 
order to reduce unnecessary traffic .on. the localvareatnetwork. Also more .preferably. the-peeridienVdistfibutes the* . 

response message by broadcast or multicast.-. •* .. ■ ':-i. .. ; ' •'' ■:: •• - . tc ■ ;■ •>!.•■•'..! .v - -. . 

[0045] < -^riexampte/as-shbwn bvRflurofcA. H--pwr'elierir-A".-reque8t8-a data package <-.W% patient f^woukl > 
reply with the response message', since peer client "B" has the data package stored locally. By contrast. -peer-client "C" 
would hot reply with a response message, since peer clienrC" does not have data package W stored Idcally On the- 
other hand, if peer client "A" requests a data package "X". both peer client "B" and peer client^ could respond In this '■•.• 
situation,, preferably only peer client "B" or peer-client "C"' would respond, depending on which' peer 'client had the" 
shorter random interval for waiting, before sending the respbiise message ' "' 

[0046] More preferably, responses are sent only for data packages with'yet unknown locations. For example, sup- 
pose client "A" requests data packages "W--. ^X"; "Y " arid "Z": Client "B-has data packages-W, "X'and "Y", and tthe ■ 
first to reply with a reply message indicating possession of data packages W, "X" and "Y". Suppose another client C • • 
has data packages "X". "Y ? and "Z": Since it replied after client "B", theresponse message from client "C" will only indi- • 
cate possessiori of data package "Z" because this.is the only data package with'an as yet unknown location. ' 
[0047] ^ - A response-message optidnallyJcbHtaihs : the'identifying PVN. the listof MD5 digests of data packages that • 
were found and a TCP portnumber. as shown in Figure 2D. The port number identifies on which TCP port the respond- •> 
ing "peer client is waiting for data packageVequests: Arterriativel* the response message optionally contains other indi- •• 
cators whfch enable the requesting.client to retrieve one or more data packages fromthe responding peer. Preferably, 
response messages are also be broadcast for data packages-which are currently being downloaded from an external 
server, for reasons described in greater detail below. •' '■ = '• ; ;": 

[0048] ■■■■■ Mn step s the peer client downloads the data package or datapackages. In principle, according to a relatively 
simple embodiments the present inventionjiafthis stage the requesting client either receives a reply and downloads 
the data packages from the client that replied; br ; ;if ■ a:reply is not received within a certain period of time, proceeds to 
download these data packages from an external server. If the peer client is downloading adata package from another 
peer client as=an internal server, the data package-exchange protocol is used to obtain the data package. The data 
package-exchange protocol is based on some appropriate peer-tc-peer communication protocol; including but not lim- 
ited 1o the "HTTP, protocol (see RFC-2068.'' "Hypertext Transfer Protocol - HTTP/1.1". available from httpV/ds-inter- < 
nir: netyrfc/rf c2068.txt as of September 23. 1 998)i ; '='••■' • . ' ' ' \._ ' " .. ''. ' • 

[0049] Preferably, a more complex implemehtatibri is employed, since such a simple implementation may cause ■ 
multiple clients to fetch the same data packages from the external server simultaneously. This situation would arise if 
several peer clients need to download the same data packages at approximately the same time, which is a very prob- 
able scenario for push clients v for Which corrteht'delivery is triggered by an external server, since none of these clients 
would receive a response to its request. Instead, the other clients would still be downloading the data package when 
the new client request is broadcast such that none of them would be ready to serve these data packages. iThus. many 
or even all of the clients would attempt to retrieve the data package from the external server and not from another peer 
client, thereby increasing the amount of traffic on the network and reducing the efficiency of operation of the system of 
th present invention.- if 

[0050] Preferably the problem is solved by notifying other clients when a first client is downloading the data pack- 
age from the external server, even if the process of retrieving the data package is not yet complete. In this preferred 
embodiment- the first ciient which reqiiires the^data package obtains the datf'package frbnvthe external server. Other 
clients which require the data package wflltheri download it from the first client even if the first client-is still in the process • 
of retrieving the data package from the external server. The preferred embodiment of the method of the present inven - 
tion is described in greater d^ ' '.. ; 
[005ir • In step 1.' the requesting client again transmits' the request, again preferably by broadcasting or multicasting; 
and then watts for a response. If no response is received within a certain period of time; in step 2 the client transmits a ; 
response message as if replying to its own- request; indicating that this client either'has the data package, or in this 
case, that the client is retrieving the data package. In step 3, the client retrieves the data package from the external 


server. • ■ • "'• • • '" • 


[0052] ' • In step *; bther clients create ari -entry in their', network data packages hash table, indicating thelocation of 
the client which will be serving the data package. Thus, preferably only a single client accesses the external server for 

any given data package. ;? • :• > " : - '" " .' • ' " . .. , 

[0053]- If a request is sent for multiple data packages; but a r esponse is received indicating the location of only some 
of the data packages at a neighboring peer client or clients; the client first obtains these data packages from the neigh- 
boring peer client or clients. Next the ciient then transmits the response message for the rest of the data packages, and 
proceeds to obtain the remaining data package or data packages from the external server. Thus; the client only obtains 
the data package or data packages from the external server which are not available locally, rather than obtaining all of 
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the data<packages frpm-the external server, thereby reducing network-.traffjc. • - r • -r : 

[0054]^??i& A6cording tcpreferred- embodiments of the present irwemibrn-preferably the" process of downloading data: ■ 
package from pe'er ctients:is optimized to reduce'trre amount of time'mquired for downloading.- the load on each indi- . 
vidual client antfthe overall'network traffic. Such optimization is'peffbrmed as follows. v, 
[0055] First, preferably the exit degree of each client is bound, such that each client is only able -to* serve a fixed. 
limited number. ol , other clients simultaneously. More, preferably, the .default limit is three other clients, for example., or- 
some another; appropriate number which is preferably.conf igured by the user or by the network administrator. If an addi- : 
tional.client attempts to download a data package from anient which is already serving themaximum number of other . 
clients will receive a -busy'. message. This feature limits the load on each individual client.. ^ : < : • .-. - . 
[0056] Also preferably, the present invention is able to optimize the selection of the.bestclient from which the data ; : 
package should be obtained. For example, if client "A" had already downloaded a larger portion of r the required data 
package than client "B". transferring the data package from client "A" ms more optimal. Such clients are preferentially «. 
selected to serve data packages, since these clients will be able to serve the data package after a shorter time period ; 
has elapsed. Such preferential selection, occurs by shortening the, time, period ;*for; waiting before these clients* respond, - 
thereby increasing the likelihood that they will serve. the data packages. For this reason; the client preferably calculates ; 
the random delay before responding such that the delay is inversely proportional .to the percentage of. the data packag / 
which has been already downloaded. In addition, : the random delay; is. preferably, proportional to the number of clients v 
being served at the.moment, in order to decrease.the likelihood of overloading already busy. clients. • ; - ; ; v -\.v- 
[0057] In addition, according to other preferred embodiments, of the.pr.esent inverttion.-pr.eferably.the entries of th < 
locations of data packages in the network data packages table are. updated according to a probabilistic function. Such, 
a function is preferred in order to prevent all of theclients from registeringa single client as the server for any particular;.- > 
data package, for example. When different clients respond, usually at. different, times, indicating they have a specific ... 
data package.,the remaining clients listening across the networkjjpdate the entry forthis.data package in thai r. network , ; 
data packages table, by adding the. IP address, or some other-type of 'address according to the addressing system; : . 
employed by the network, of the client which can serve the data packagejto this table. In a simple implementation; -the-, 
clients would .store only the last advertised location of each data :Package;--and therefore: many or all: clients might.,;: 
attempt to obtain the data package from a single client as the internaljseryer.-thereby overloading>that client. : ^- . v<». 
[0058] To.ayord this situation, preferably the following. probabilistic -algorithm . is. used to determine the particular .cli- iv 
ent address which is stored in the network data packages table. Each time, a new client transmits a response message,;., 
indicating that this client is able to serve a particular data package, the probability that the new, IP addresspf the ; new . : . 
client.is substituted for. the old IP address is calculated according to-.th^ifpllpwing, equations:. .-, ». . • • :n- ..« \ : * y±**.\ : r> 


; New IP address - < 


New BP address Pn(x) = 1 - l/(generation+ 1 ) ... 


45 


SO 


55 


wherein Pn(x) is the probability that anew IP address. is substrtuted.fo^thepl.d. IP address, Pp(x), is the probability that . 
the old IP address is retained, . and "generation" is a number indicating how. many times, this address, had; been p.revi-: ■ 
ously changed. y. . . r n-. - . - . i 

[0059] For example, if client "A" responds indicating it has data package *X vtheniinitially all other peer clients store, 
the IP address of client "A" as the location of data package "X", If client; B" then broadcasts a response also indicating 
that client -B" has data package, H X\ then the probability that any one client now changes the IP address for the location 
of data package "X" is 50%. In other words.. about half of the clients should now,ppint to client "A" and about half should 
pointto:client "B"., : : : - . . t t ^- .'. v."' 1 

[0060] Such a substantially even distribution of load across multiple clients should produce data-flow with a tree- 
shaped topology, as shown in Figure 3, rather than a randornjopology. thus optimizing the average downloadtime and., 
the load on the serving clients. _ . ..«.--\v.£' * 1 * ; : •> •-. 

[0061] Furthermore, if any client requests a particular data package during the period required by cl|ent"A".for 
downloading that package, preferably client "A" sends a broadcast er- multicast message indicating that the package is 
in the process of.being downloaded; Therefore, preferably only a single client "B" polls client "A" for each data package, 
for example. Other clients preferably automatically receive any responses from that polling action though the broadcast 
or multicast transmission, and thus will not be forced to poll for themselves. ... * - . 
[0062] The polling (request/response) traffic is optimized, since there is usually norieed to transmit both a request 
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and a response for each data package needed : by'eac^client:Such optimizationis possible sirice each^lieW^efefag 
receivesJubstantially all of the request/respbhseveomWunication c4-anihe-c*er^^?feme^ . 
of the data packages ; in the network<latapackages ! :table:.J : •■ ' ■ : - •■';■;"• ^;*\". .7.. .. 

[0063] • • As-previously described; te^pm******** '^^^^'^^T^S^ifS^- 
Luri Wring to thedata package exchahge prAo'coirfay uang the HTTP prot^ol or some ofrer wjgtepejr^- 
peer communication protocol. The data package transfer software application ofthe present , 
Cures' a timer/for detection of an aborted transfer or aVery slow data package transfer, for example: The timer deter- 
mines^' such a transfer has timed out. If a; time-out occurs, the requesting client. P"^*^*^ 
orocess If the VansfeV remains unsuccessful after "a' plurality of attempts, the chert preferably ceases to attempt to 
transfer the data package from the' peer client 'as the internal server.-and instead transfers the data package or date; 
Dackages directly from the external server.?' •• ' v "' . ■•.^^.^ „.kW. 

r0064] ' Again as desCTibed previously 
SenLrSShgclLreceL^ 

Sa^ibnot the dita package already dowhlbadedl The requesting client continues poll.ng the servmg client unt.. the 
data package download is complete. If the Bbwn load becomes substantially slower or is otherw.se .nterrupted or term.- 
nat d for a long period of time, thb requesting client behaves as if a timeout occurred .„^„„-„ oor 
ro065i - According to additional preferred features bf the present invention; substantially automatic detection of peer 
dienti is supported: Such Automatic detection" enables each peer client to detect the presence .of other ^*j£on , 
fhe nertwonl Ksuch peercliehts are not foUnd/preferably the system of the present .nverrtion 'sd.sabled smcethe oper- 
S sysWm Jis described above would only prolong the time period required to download a data package rf no 

XP'-pS bandwidth on the local area network which is consumed by each peer client serv- ' 

ingdata'packages to Mother clients islimited. to avoid over-burdening any specHichc^t. TWs hm 

blebvthe user or by the n^ork administratior: f .. .. ^ > iu ' +k« ■ 

SoBTl " 1 Furthermore, in order to protect peer clier^ ^ 

sSem of the breserit : invention, certain- sbcuritf features are preferably included. For example, preferably only data 
£££ idertS hThash febles are able to be transferred from the client Thus, transmitted data packages am 
p^^ortydfi packages which Were intended to be served to the.peer clients, such that mahcous users ^refera- 
SlcWSS-sy\t«h?« the present invention fe obtain 'random" data packages from the storage med^oj a :peer 
Slert^all^^wrtore ^eferabry bhiyVeferenced'by their unique Certifier, such as their 1.28*rtM05 digest. 
SiKffi^?e^. to be dbwrilbaded from a client if the intended recipient knows this Jgest Thus,; • 

W$*? : Abating to anbtheV embodiment^ the'^resent invention, the system of trie pV«ent invention is also apgP 
caWe to Web^owsers. FTP clients, and other software applications involving dient-server data-transfer. As descnbed 
S ? refbrencb'tbFig\ires^ 

SS 1 - • Mh step r of Figure 4. a Web browser beftg operated by a client computer f*^'it^t^Jfc<*W*? 
Fir* the Web browser loo* at the local cachel as is^bwn to one of ordinary skJ.I in the art. If the data package » 1w* 

sage requesting this data package, preferably by using broadcast or multicast message ^^^^^ 
aae is preferably uniquely defined by a unique identifier. More preferably, the unique .dentrf.er .s the URL of the data 
^^or^ernJ^ and preferably a combination of the URL of the data package and timestamp. or by any other 

^"To^Sak; if more tW one data 'package is required, the Web browser preferaUy t^te one 
equest message containing the list of needed data packages.thereby reducing the tote Stl^Tk^ fen- 
work. Such a situation may arise if, for example, the Web browser had just parsed an HTML (hypertext nwk-^lan 
Tuage) document or Web page, which contains many links to follow, Preferably and opt.onally. each request message 
tSS^Zi^ "magilnumber". Which may contain the protocol version (PVN). 

in Figure 5A. the request message includes' the list of URL's' br other urwqiie .denW.ers to .denWy the date package or 
data^ackages being requested, which is similar in fbnction to the list of MD5 digests described previously for request 
messages, and a unique identifier identifying the request message, shown as "REQ".. . '■ ; ' ^.^u 
[00711 In step 2. other Web browsers across the network listen to detect request messages of this type. These Web 
browsers, which are peer diehts for this erribodirneht of the present invention, receive this request message and check 
their own cache for the requested 'URL" If the requested URL is found in the local cache ofa Web browser, that Web 
browser preferably waits a random interval and then preferably transmits a response message indicating « has the 
required data package (or data packages) . Preferably, the message is broadcast or multicast. More preferably, that Web 
browser does not reply if another Web browser had replied first. A reply message is preferably sent by a particular Web 
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browse^ ev t en rf the reguested URL is .still being downloaded by ^a^Web^rpwser.. 

[0072];.:>f i ,! Jh;step;3,.t1 -no. response to an issued request message-isiregeiv.ed within a certain amount of time, for exanv 
ple 5 seconds, then the process is preferably timed-out. In this case,.;tbe;Web browser.preferably.no longer attempts.-to . 
obtain the.URL from : another,Web browser, and the URL is obtained from.the regular Web server using regular HTTP 

5 protocol. Before starting to download the data package from the regular Web server,, the Web browser, preferably trans-, 
mits.a responsemegsage indicating. that this particular Web browser is downlo^ ..... 
[0073] On the.otherhand, if a response message is received, the Web browser pbtajns the URL from the other Web 
browser which indicated that it had the URL in the local cache. Preferably, Web browsers across the network recprd the 
URLs and the. address, from, which the response message originated- for future use. such. that these Web.browsers.. 

w would be.able to.downjoad the URL at a future time- without first transmitting the reque^ - . t . 

[0074] Once the Web browser is able to locate a data package on a neighboring Web browse, the, Web browser 
attempts to download the data package. The downloading. process, is performed .with a suitable data-transfer protocol, 
such as HTTFf: or FTR : If a time-out or other failure occurs during the. processing-of data package transfer, the receiving,,, 
Web browser preferably performs substantially .the entire procedure^ rrujfe/than onc^. More, preferably, the number of . 

75 permitted attempts to retry the transfer is .configurable. If the process-fails after these attempts have been performed,. , 
preferably the Web browser transfers the required data package or data packages from, the. regular Web server. . 
[0075] According to preferred features of this- embodiment of the present. invention, .data package downloading, is . 
well distributed, such that the Web browsers :do not obtain a data package; from, only a single : Web browser, but rather , 
obtain the data package from a plurality of Web browsers. ,Such distribution is maintained as follows. 

20 [0076] First, preferably the number of simultaneous data package ,transfers from a single Web browser is. limited. If 
this number is exceeded, the Web browser transmits a "busy" message to other Web browsers attempting to transfer., 
the data package. Next, preferably once a Web browser receives a. message giving, the. location of a particular data u 
package. t the corresponding entry in the hash table for, that data package is not altered, every, time another, response 
message is received pertaining to this data package. The hash table is preferably altered by subsequent messages in ■ 

25 a probabilistic manner, such that the.probability that any particular entry is updated .to indicate, a new location of a data . 
package is equal to 1/(generation+1 ). where 'generation' counts the number of f times a response mess^ge : was|.received 
for that data package. -.' . , -wiv- -.- ■ j ;/ • .• 'v. i • . v 

[0077] . For .example, if Web browser "A" transmits a response message -indicating that data package "X" is oh the 
local cache, then initially all of, the neighboring Web browsers have an . entrain the hash table t indicating^that; Web , 

30 browser. "A" is the location of data package M X M . If Web browser "B^then ^ transrpits a response message for : data packr 
age/X"! then each Web browser preferably now alters the entry, in ; tfie,hash table to indicate a new location of data 
package "X". with a probability of about fifty percent, such .that abQuUjfjx. percent of. the Web ^rowsers : now : .have ap . 
entry indicating that the data package is available from Web browser "A" and such that about fifty percent of the Web . ( 
browsers.now have an^entry indicating that the data package is. availably frpm Web browser :B W . Thus, a good load.di^-. ; 

35 tribution can be achieved. , . ' ... . 

[0078] '.'..the random delay (mentioned. in step 2 above) chosen r (?y.abrpwser..is proportional : to the number, of cur-, 
rently served "browsers, or the number of browsers currently downloading data packages from that browser* and^_ 
inversely proportional to.the. amount of thg data package already dqwnj[qaded.by it. Tfiis way the t browsers more eligfcie. : 
to : download from are, more likely .to be. chosen by other browsers to .serve these data packages, ...... 

40 [0079] ' v . While the invention has been described with respect to : a| : ijmited/number of embodiments," it will be appreci- r 
ated that rpany variations, mpdif ications. and other applications of the.inyention may be made. , . ' . r . : 

Claims. '' *. t . *. . '. / ' ^ f , ^r-' ! vo« : «..^-/- *■*.'■-■ * 

45 1 . . A method for distributing data packages across a network, the network featuring an external server for* serving at . 
least one data package, the external server^being a dedicated ^rver ( the ( .steps of the method being performed by 
. a data processor, the method comprising the steps of: . ... • ... t , . ,j 

(a) providing a plurality of peer clients attached to.the network and providing, a list of data packages,, 6aid data 
so packages being stored by each of said plurality. of peer clients, each data package of said data packages hav- 

ing an entry in said list, said entry indicating a unique identif ieriorj^aid data.packiage gnd a lopatipn of said data 
package in at least one of said plurality of. peer clients; , ; 

. , , . (b) examining said list ol data packages by a first peer client to. find an entry for a required. data package; and 
. (c). if said entry for said data package is present on said list of data packages of said first peer client, retrieving 
55 said data package from said location at another of said plurality of peer clients according to said entry for said 

. data'package. ' ... 

2. The.method of. claim.'l , wherein said list of data packages is stored on at least said first peer client. 
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3 The method of claim 2 , wherein alternatively said entry for said data package is absent from said list of data pack- 
ages of said first peer client, the method further comprising the steps of: 

(d) sending a request message for said data package by said f irst peer client to at least one other peer client; 

(eTif a response message is received by said first peer client from said at least one other peer client, retrieving 
said data package from said at least one other peer client by said fret peer client. 

4. The method of claim 3. the method further comprising the step of: 

(f) altering said list of data packages being stored by at least said first peer client tor indicating said location of 
said data package according to said response message. 

5 The method of claim 4. wherein if said response message is not received from said at least one other peer dent 
by said first peer client, the method further comprises the step of: 

(g) obtaining said data package by said first peer client from the external server. 

server. 

7. The method of daim 6, wherein said list of data packages is stored on each of said plurality of peer clients, the 

method further comprising the steps of: 

tion of said data package according to said response message. , ; 

30 8. The method of daim 5. wherein said list ofdata packages is stored on eacK of said plurality of peer clients, the 

method further comprising the steps of: 

tion of said data pad^aowrdihg to a probaWnstic fundw. f ; ' ■■ ' .' 

- j . J. , J , » r " ' 1 '''Tr 

method further comprises the step of: 

(d) sending a busy message from said at least,one other peer client to said first peer client. 

10. The method of claim 1. wherein the external server is a BackWeb- server, and said plurality of peer clients is a 
plurality of BackWeb™ clients. 

11. Asystemfor distributing data packages across a network according to a list of the data packages, the system com- 
prising: 

(a) an externa, server for serving at least one data Wti* said external server being attached to the network: 

to? a plurality of peer clients attached to the network, the data packages being stored by each of said plurality 
S oee S i £5 date package of said data packages having an entry in the l.st. sad entry ■ ,nd catng a 
unioue idSer^said date parage and a location of said data package in at least one of sa.d plurality of 
oee ^ ients s^cMh? each peer client retrieves a data package according to the list each peer Cert I «l 
VZ^mtoMtonJu date package from another of said plurality of peer clients, and attentively retr.ev- 
?ng S diS froJ fsSd'externa. server H said data package is not available from another of sa«f plu- 
rality of peer clients. 
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